Assessing clinical performance in cardiac surgery. Does a specialised clinical database make a difference
http://www.100md.com
《血管的通路杂志》
a Department of Clinical Governance, Agenzia Sanitaria Regionale, Viale Aldo Moro, 21, 20147 Bologna, Italy
b Division of Cardiology, Azienda Ospedaliera, Parma, Italy
c Department of Cardiac Surgery, Hesperia Hospital, Modena, Italy
d Department of Cardiac Surgery, Villa Maria Cecilia, Lugo, Ravenna, Italy
e Department of Cardiac Surgery, Azienda ospedaliera, Bologna, Italy
f Department of Cardiac Surgery, Villa Salus, Reggio Emilia, Italy
Abstract
We compared mortality rates league tables for six cardiac surgery centres developed using an administrative database (integrated with information on patients' EuroSCORE) with those drawn from a specialised clinical database. Data from 4017 patients undergoing cardiac surgery over the period January 1st–December 31st 2003, and identified both databases were used. Case mix adjusted in-hospital mortality rates were estimated relying on information provided by each database, and league tables were drawn from both. The correlation between the two league tables was assessed through the Spearman correlation coefficient. League tables drawn from the two sources identified the same ‘best’ and ‘worst’ centres and the Spearman correlation coefficient confirmed a high level of agreement between the two rankings (r=0.89; P<0.02). Use of the logistic EuroSCORE instead of the additive one did not change the results. An administrative and a clinical specialised database provided similar league tables. However, this finding by no means implies that clinical databases should be abandoned. While administrative data allow a more efficient performance assessment, clinical databases may more properly satisfy the legitimate demand of surgical staff of being directly involved in quality monitoring, rather than being mere passive objects of external assessment.
Key Words: Administrative data; Clinical database; Cardiac surgery; League table; Case-mix adjustment
1. Introduction
Comparative assessment of hospital performance, usually in the form of league tables, is currently one of the most frequently addressed topics in health services research, and surely one of those relevant to cardiac surgery. When providers are compared, a number of problems arise concerning the statistical techniques used in outlier identification and the broad impact that these initiatives may have on individual hospitals and their staff [1].
Leaving the policy implications of public disclosure aside [2], one of the technical issues at stake is the relevance of the source of information on which the performance assessment is based. Indeed, information source's role is crucial for the assessment of clinical outcomes, as detailed information is needed if individual patient's complexity and surgical risk has to be properly taken into account.
While offering the obvious advantage of being easily available at low cost, administrative databases are by their very nature usually considered inadequate when it comes to adjusting for case-mix [3]. Thus, specialised clinical databases, based upon a prospective collection of detailed information on clinical characteristics of individual patients, could be considered a valuable alternative. However, their organisation and management can be complex and resource demanding.
Both those alternatives are currently being experimented in Emilia-Romagna, an Italian region of 4 million residents with six cardiac surgery centres.
In this regional context, the system of continuous monitoring of hospitals' clinical performance, which has been in place since 1998 [4], relies on the regional administrative database of hospital admissions (SDO) as the main source of information for the assessment of mortality rates for any type of cardiac surgery. A remarkable feature of this system is the inclusion of a description of patient's surgical risk expressed through the EuroSCORE [5]. This information has no administrative value and it is included only for quality assessment purposes.
On April 2002, as part of a broader effort to improve our quality monitoring system, a regional registry of patients undergoing cardiac surgery was launched, with individual centres required to collect detailed information on patient characteristics and outcomes. In undertaking this initiative, it was assumed that this more clinically oriented source of information would have improved case-mix adjustment procedures, thus providing more accurate mortality figures for each centre and, overall, a fairer assessment of their performance. The coexistence of the two databases made it possible to assess the extent to which that assumption held true.
In this paper we compare mortality rates league tables for six cardiac surgery centres, developed using the administrative database of hospital admissions and the specialised clinical database (the registry), respectively.
In particular, the aim was to assess whether the additional information provided by the specialised clinical database had any impact on the performance assessment of the individual centres, as compared to the same effort undertaken simply relying on the administrative data integrated with information on patients' EuroSCORE.
2. Materials and methods
2.1. General characteristics of the databases
The regional hospital admissions database (SDO) was established in 1991, and since 1995 it has been routinely used for administrative purposes, including the attribution of each individual episode of care to the corresponding DRG. Primary and secondary diagnoses, as well as diagnostic and surgical procedures are recorded using the ICD-9-CM classification system. In short, information on patient age, gender, co-morbidity, complications and vital status at discharge is available. For patients undergoing cardiac surgery, information on their surgical risk according to the EuroSCORE is also available.
In April 2002, a specialised registry for cardiac surgery procedures was established, with the aim of providing individual centres with detailed feedback on their activity and on the clinical outcomes of their patients. The registry collects more detailed information on the clinical characteristics of patients undergoing cardiac surgery, and on those of the surgery performed.
While both sources include the EuroSCORE, this information is reported differently. While in the administrative database only the overall score attributed to each patient by the surgical staff is explicitly reported, in the clinical database each clinical characteristic concurring with the EuroSCORE calculation is reported, and the score is then estimated centrally.
2.2. Statistical analysis
For the purposes of this analysis, 4017 of 4175 patients undergoing cardiac surgery at one of the 6 regional centres over the period January 1st–December 31st 2003 were considered. Patients identified in the SDO database but not in the registry were excluded (n=158).
Case-mix adjusted in hospital mortality rate for each centre was assessed using both administrative and clinical data.
Adjustment was performed using logistic regression models, applied separately to each source of information.
In particular, the relationship between death and patient clinical characteristics was assessed by univariate analyses, and variables emerging as statistically significant were then included in a logistic regression model, whose calibration and prediction were assessed with the Hosmer-Lemeshow test, and the c statistic, respectively [6].
The results of the two models were used to estimate individual patient's likelihood of death, and the overall number of expected events for each centre.
To evaluate the statistical significance of the difference between observed and expected events (O-E), the standard error of the observed rate was calculated according to the following formula: [P x (1–P)/n]1/2, where n is the total number of patients, and P is the expected mortality rate. Then a z statistic was calculated, dividing O-E by the standard deviation [7]. When z is –1.96 or 1.96, the difference can be considered statistically significant at a P-value 0.05.
Risk-adjusted death rates were then calculated by multiplying the observed/expected ratio by the overall regional mortality rate over the study period. The two different league tables thus obtained ranked centres according to the risk-adjusted rate estimated.
In addition, the correlation between the two rankings was assessed with the Spearman correlation coefficient.
Case volume of individual centres (overall and by type of surgery) and crude in-hospital mortality rates are reported in Table 1.
3. Results
A description of the general characteristics of the 4017 patients considered, as drawn from the two databases, is reported in Table 2.
Prevalence of risk factors (hypertension, diabetes, obesity) was higher when assessed in the clinical database. As for the surgical risk according to EuroSCORE, 67% of the patients scored 6 (i.e. high surgical risk) in the clinical database, while the same figure drawn from the SDO was 48% (Table 2).
Table 3 outlines the best fitting logistic regression models drawn from the two databases. As shown, while the SDO model included only EuroSCORE and type of surgery, the one applied to the clinical database additionally identified two variables not considered in the administrative database (dialysis and priority). Nevertheless, the two models did not differ in terms of predictive ability, as expressed by the c statistic (78% vs. 77%).
The implications of the use of the two regression models for estimating case-mix adjusted in-hospital mortality rates for individual centres are reported in Table 4.
Adjusted mortality rates estimated from the two approaches differed substantially for two centres (by 1.9% and 1.6% for centres C and A, respectively), but were similar in the remaining cases. Nevertheless, one hospital (A) was identified by both models as a negative outlier (i.e. observed events exceeded the number predicted by both models, and the difference was statistically significant), while another (D) emerged as a positive outlier (i.e. number of observed events was significantly lower than expected) in the SDO analysis, but not in the clinical database analysis.
League tables drawn from the two approaches are shown in Table 5. In both cases, the same ‘best’ (i.e. rank 1) and ‘worst’ (i.e. rank 6) centres were identified. Spearman correlation coefficient confirmed a high level of agreement between the two rankings (r=0.89; P<0.02). The use of the logistic EuroSCORE in alternative to the additive one did not change the result.
4. Discussion
Exploring the implications of assessing clinical performance through different sources of information is not merely a technical exercise. Given the current emphasis on health care providers' public accountability and on the value of performance indicators as a policy tool for improving quality of care, it is highly relevant to explore the extent to which those goals could be achieved by simply relying on sources of information already available or, alternatively, by developing specific and much more demanding tools.
Thus, it is not surprising that this issue has been the object of a number of studies, especially in the area of cardiac surgery [8–13]. However, the results are quite conflicting, ranging from those convincingly supporting the use of administrative databases for quality assessment purposes [9,10,12], to those claiming the superiority of clinical databases when it comes to developing mortality rates report cards or league tables [8,11,13].
According to our findings, individual centre mortality rates obtained from both sources do show some differences. For two centres in particular, adjusted mortality rates differed by about 2% (Table 4). However, the practical relevance of those discrepancies was minimal, as the same centres were identified as outliers in both databases, and had no substantial effect on their ranking.
The same findings held true when the analyses were performed separately by type of surgical procedure: as shown in Table 6, rankings of case-mix adjusted mortality rates for isolated coronary artery bypass graft drawn from both databases were strikingly similar. In addition, the same degree of consistency between clinical and administrative database was confirmed when the analysis was repeated considering 30-day, rather than in-hospital, mortality, and using data from year 2004 (data not shown).
Nevertheless, it is questionable whether outlier identification relying on mortality rates should be the only goal of performance monitoring. Monitoring the performance of cardiac surgery centres may well focus also on outcomes other than mortality, and should be done in such a way that surgical staff are not just passive objects of the evaluation, but actively involved in the assessment process.
In this perspective, clinical databases may have an obvious advantage over administrative ones. As shown in our study, they provide more complete information on outcomes overlooked by administrative data (i.e. complication rates) and, by their very nature, require the active participation of surgical teams in the data collection process. Another advantage offered by a clinical database is the opportunity to rely on logistic rather than additive EuroSCORE in case mix adjustment. Indeed, the logistic EuroSCORE has been shown to perform better than the additive one in predicting mortality in high risk patients [14]. However, reliance on logistic EuroSCORE did not provide any difference in the ranking of our six centres (see Table 5).
In comparing administrative vs. clinical databases, it is worth noting that the former can be partially modified to allow their exploitation for the assessment of clinical performance. In Emilia-Romagna region, the administrative database of hospital admissions (SDO) has the peculiar feature of including key clinical information not relevant to administrative purposes, but deliberately collected to be used for assessing clinical performance taking into account patient complexity. In fact, since 1998, in cardiac surgery centres, the information describing individual patient characteristics in the SDO database includes the surgical risk profile expressed with the EuroSCORE system.
As others have already pointed out [10], a key issue supporting the value of clinical databases is their greater face validity to health care providers and health professionals. Indeed, if the objective of clinical performance monitoring is the development of report cards to be made available to the public, the degree of trust that health professionals place on the source of information used for the assessment has a crucial role in making the initiative acceptable.
Nevertheless, although it is important, source of information is just one of the aspects affecting the credibility of performance monitoring initiatives for health professionals, and not necessarily the most relevant. Others include the extent to which the main objectives of these efforts are shared with health care providers, the relevance of the indicators used in monitoring clinical performance and, in general, the extent to which these efforts are perceived by health professionals to be supportive rather than punitive [15].
5. Conclusions
In ranking cardiac surgery centres according to their case-mix adjusted mortality rates, an administrative database including key clinical information provided the same results as a more complex and resource-demanding specialised database. However, as in the case of our study, showing that inexpensive administrative databases may reach the same conclusions as clinical databases, in labelling individual hospitals as ‘outliers’, does not mean that specialised clinical databases should necessarily be abandoned.
While there is a large consensus on the need for monitoring hospital performance, it should be noted that the choice of a source of information over another implicitly implies also a choice between different approaches to quality improvement and models of accountability [15]. In such a context, the main added value of clinical databases is the degree of direct participation of surgical teams required by their implementation. Such participation represents the necessary premise to expanding the scope of performance monitoring beyond the mere identification of ‘bad apples’ by some external agency, rather emphasising quality assessment as a specific responsibility of individual surgical teams.
Acknowledgements
Funding: support for this study was provided by Agenzia Sanitaria Regionale, Bologna (Italy).
References
Marshall M, Shekelle P, Brook R, Leatherman S. Dying to know: public release of information about quality of health care 2000;London (UK): Nuffield Trust and RAND.
Treasure T. Mortality in adult cardiac surgery. Br Med J 2005; 330:489–490.
Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997; 127:666–674.
Nobilio L, Fortuna D, Vizioli M, Berti E, Guastaroba P, Taroni F, Grilli R. Impact of regionalisation of cardiac surgery in Emilia-Romagna, Italy. J Epid Community Health 2004; 58:97–102.
Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999; 16:9–13.
Hosmer DW, Lemeshow S. Applied Logistic Regression 1989;New York: John Wiley & Sons.
Goldfield N, Pine M, Pine J. Measuring and managing health care quality. Procedures, techniques, and protocols 1991;Gaithersburg, MD: Aspen Publishers.
Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcomes measures and data sources. Am J Pub Health 1994; 84:1609–1614.
Geraci JM, Johnson ML, Gordon HS, Petersen NJ, Shroyer AL, Grover FL, Wray NP. Mortality after cardiac bypass surgery. Prediction from administrative versus clinical data. Med Care 2005; 43:149–158.
Ghali WA, Rothwell DM, Quan H, Brant R, Tu JV. A Canadian comparison of data sources for coronary artery bypass surgery outcome report cards. Am Heart J 2000; 140:402–408.
Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough. Health Serv Res 1997; 31:659–678.
Landon B, Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD. Judging hospitals by severity-adjusted mortality rates: the case of CABG surgery. Inquiry 1996; 33:155–166.
Hannan EL, Kilburn Jr H, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery. Does it matter. Med Care 1992; 30:892–907.
Shanmigam G, Wets M, Berg G. Additive and logistic EuroSCORE performance in high risk patients. Interact CardioVasc Thorac Surg 2005; 4:299–303.
Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001; 72:2155–2168.(Daniela Fortuna, Maria Vi)
b Division of Cardiology, Azienda Ospedaliera, Parma, Italy
c Department of Cardiac Surgery, Hesperia Hospital, Modena, Italy
d Department of Cardiac Surgery, Villa Maria Cecilia, Lugo, Ravenna, Italy
e Department of Cardiac Surgery, Azienda ospedaliera, Bologna, Italy
f Department of Cardiac Surgery, Villa Salus, Reggio Emilia, Italy
Abstract
We compared mortality rates league tables for six cardiac surgery centres developed using an administrative database (integrated with information on patients' EuroSCORE) with those drawn from a specialised clinical database. Data from 4017 patients undergoing cardiac surgery over the period January 1st–December 31st 2003, and identified both databases were used. Case mix adjusted in-hospital mortality rates were estimated relying on information provided by each database, and league tables were drawn from both. The correlation between the two league tables was assessed through the Spearman correlation coefficient. League tables drawn from the two sources identified the same ‘best’ and ‘worst’ centres and the Spearman correlation coefficient confirmed a high level of agreement between the two rankings (r=0.89; P<0.02). Use of the logistic EuroSCORE instead of the additive one did not change the results. An administrative and a clinical specialised database provided similar league tables. However, this finding by no means implies that clinical databases should be abandoned. While administrative data allow a more efficient performance assessment, clinical databases may more properly satisfy the legitimate demand of surgical staff of being directly involved in quality monitoring, rather than being mere passive objects of external assessment.
Key Words: Administrative data; Clinical database; Cardiac surgery; League table; Case-mix adjustment
1. Introduction
Comparative assessment of hospital performance, usually in the form of league tables, is currently one of the most frequently addressed topics in health services research, and surely one of those relevant to cardiac surgery. When providers are compared, a number of problems arise concerning the statistical techniques used in outlier identification and the broad impact that these initiatives may have on individual hospitals and their staff [1].
Leaving the policy implications of public disclosure aside [2], one of the technical issues at stake is the relevance of the source of information on which the performance assessment is based. Indeed, information source's role is crucial for the assessment of clinical outcomes, as detailed information is needed if individual patient's complexity and surgical risk has to be properly taken into account.
While offering the obvious advantage of being easily available at low cost, administrative databases are by their very nature usually considered inadequate when it comes to adjusting for case-mix [3]. Thus, specialised clinical databases, based upon a prospective collection of detailed information on clinical characteristics of individual patients, could be considered a valuable alternative. However, their organisation and management can be complex and resource demanding.
Both those alternatives are currently being experimented in Emilia-Romagna, an Italian region of 4 million residents with six cardiac surgery centres.
In this regional context, the system of continuous monitoring of hospitals' clinical performance, which has been in place since 1998 [4], relies on the regional administrative database of hospital admissions (SDO) as the main source of information for the assessment of mortality rates for any type of cardiac surgery. A remarkable feature of this system is the inclusion of a description of patient's surgical risk expressed through the EuroSCORE [5]. This information has no administrative value and it is included only for quality assessment purposes.
On April 2002, as part of a broader effort to improve our quality monitoring system, a regional registry of patients undergoing cardiac surgery was launched, with individual centres required to collect detailed information on patient characteristics and outcomes. In undertaking this initiative, it was assumed that this more clinically oriented source of information would have improved case-mix adjustment procedures, thus providing more accurate mortality figures for each centre and, overall, a fairer assessment of their performance. The coexistence of the two databases made it possible to assess the extent to which that assumption held true.
In this paper we compare mortality rates league tables for six cardiac surgery centres, developed using the administrative database of hospital admissions and the specialised clinical database (the registry), respectively.
In particular, the aim was to assess whether the additional information provided by the specialised clinical database had any impact on the performance assessment of the individual centres, as compared to the same effort undertaken simply relying on the administrative data integrated with information on patients' EuroSCORE.
2. Materials and methods
2.1. General characteristics of the databases
The regional hospital admissions database (SDO) was established in 1991, and since 1995 it has been routinely used for administrative purposes, including the attribution of each individual episode of care to the corresponding DRG. Primary and secondary diagnoses, as well as diagnostic and surgical procedures are recorded using the ICD-9-CM classification system. In short, information on patient age, gender, co-morbidity, complications and vital status at discharge is available. For patients undergoing cardiac surgery, information on their surgical risk according to the EuroSCORE is also available.
In April 2002, a specialised registry for cardiac surgery procedures was established, with the aim of providing individual centres with detailed feedback on their activity and on the clinical outcomes of their patients. The registry collects more detailed information on the clinical characteristics of patients undergoing cardiac surgery, and on those of the surgery performed.
While both sources include the EuroSCORE, this information is reported differently. While in the administrative database only the overall score attributed to each patient by the surgical staff is explicitly reported, in the clinical database each clinical characteristic concurring with the EuroSCORE calculation is reported, and the score is then estimated centrally.
2.2. Statistical analysis
For the purposes of this analysis, 4017 of 4175 patients undergoing cardiac surgery at one of the 6 regional centres over the period January 1st–December 31st 2003 were considered. Patients identified in the SDO database but not in the registry were excluded (n=158).
Case-mix adjusted in hospital mortality rate for each centre was assessed using both administrative and clinical data.
Adjustment was performed using logistic regression models, applied separately to each source of information.
In particular, the relationship between death and patient clinical characteristics was assessed by univariate analyses, and variables emerging as statistically significant were then included in a logistic regression model, whose calibration and prediction were assessed with the Hosmer-Lemeshow test, and the c statistic, respectively [6].
The results of the two models were used to estimate individual patient's likelihood of death, and the overall number of expected events for each centre.
To evaluate the statistical significance of the difference between observed and expected events (O-E), the standard error of the observed rate was calculated according to the following formula: [P x (1–P)/n]1/2, where n is the total number of patients, and P is the expected mortality rate. Then a z statistic was calculated, dividing O-E by the standard deviation [7]. When z is –1.96 or 1.96, the difference can be considered statistically significant at a P-value 0.05.
Risk-adjusted death rates were then calculated by multiplying the observed/expected ratio by the overall regional mortality rate over the study period. The two different league tables thus obtained ranked centres according to the risk-adjusted rate estimated.
In addition, the correlation between the two rankings was assessed with the Spearman correlation coefficient.
Case volume of individual centres (overall and by type of surgery) and crude in-hospital mortality rates are reported in Table 1.
3. Results
A description of the general characteristics of the 4017 patients considered, as drawn from the two databases, is reported in Table 2.
Prevalence of risk factors (hypertension, diabetes, obesity) was higher when assessed in the clinical database. As for the surgical risk according to EuroSCORE, 67% of the patients scored 6 (i.e. high surgical risk) in the clinical database, while the same figure drawn from the SDO was 48% (Table 2).
Table 3 outlines the best fitting logistic regression models drawn from the two databases. As shown, while the SDO model included only EuroSCORE and type of surgery, the one applied to the clinical database additionally identified two variables not considered in the administrative database (dialysis and priority). Nevertheless, the two models did not differ in terms of predictive ability, as expressed by the c statistic (78% vs. 77%).
The implications of the use of the two regression models for estimating case-mix adjusted in-hospital mortality rates for individual centres are reported in Table 4.
Adjusted mortality rates estimated from the two approaches differed substantially for two centres (by 1.9% and 1.6% for centres C and A, respectively), but were similar in the remaining cases. Nevertheless, one hospital (A) was identified by both models as a negative outlier (i.e. observed events exceeded the number predicted by both models, and the difference was statistically significant), while another (D) emerged as a positive outlier (i.e. number of observed events was significantly lower than expected) in the SDO analysis, but not in the clinical database analysis.
League tables drawn from the two approaches are shown in Table 5. In both cases, the same ‘best’ (i.e. rank 1) and ‘worst’ (i.e. rank 6) centres were identified. Spearman correlation coefficient confirmed a high level of agreement between the two rankings (r=0.89; P<0.02). The use of the logistic EuroSCORE in alternative to the additive one did not change the result.
4. Discussion
Exploring the implications of assessing clinical performance through different sources of information is not merely a technical exercise. Given the current emphasis on health care providers' public accountability and on the value of performance indicators as a policy tool for improving quality of care, it is highly relevant to explore the extent to which those goals could be achieved by simply relying on sources of information already available or, alternatively, by developing specific and much more demanding tools.
Thus, it is not surprising that this issue has been the object of a number of studies, especially in the area of cardiac surgery [8–13]. However, the results are quite conflicting, ranging from those convincingly supporting the use of administrative databases for quality assessment purposes [9,10,12], to those claiming the superiority of clinical databases when it comes to developing mortality rates report cards or league tables [8,11,13].
According to our findings, individual centre mortality rates obtained from both sources do show some differences. For two centres in particular, adjusted mortality rates differed by about 2% (Table 4). However, the practical relevance of those discrepancies was minimal, as the same centres were identified as outliers in both databases, and had no substantial effect on their ranking.
The same findings held true when the analyses were performed separately by type of surgical procedure: as shown in Table 6, rankings of case-mix adjusted mortality rates for isolated coronary artery bypass graft drawn from both databases were strikingly similar. In addition, the same degree of consistency between clinical and administrative database was confirmed when the analysis was repeated considering 30-day, rather than in-hospital, mortality, and using data from year 2004 (data not shown).
Nevertheless, it is questionable whether outlier identification relying on mortality rates should be the only goal of performance monitoring. Monitoring the performance of cardiac surgery centres may well focus also on outcomes other than mortality, and should be done in such a way that surgical staff are not just passive objects of the evaluation, but actively involved in the assessment process.
In this perspective, clinical databases may have an obvious advantage over administrative ones. As shown in our study, they provide more complete information on outcomes overlooked by administrative data (i.e. complication rates) and, by their very nature, require the active participation of surgical teams in the data collection process. Another advantage offered by a clinical database is the opportunity to rely on logistic rather than additive EuroSCORE in case mix adjustment. Indeed, the logistic EuroSCORE has been shown to perform better than the additive one in predicting mortality in high risk patients [14]. However, reliance on logistic EuroSCORE did not provide any difference in the ranking of our six centres (see Table 5).
In comparing administrative vs. clinical databases, it is worth noting that the former can be partially modified to allow their exploitation for the assessment of clinical performance. In Emilia-Romagna region, the administrative database of hospital admissions (SDO) has the peculiar feature of including key clinical information not relevant to administrative purposes, but deliberately collected to be used for assessing clinical performance taking into account patient complexity. In fact, since 1998, in cardiac surgery centres, the information describing individual patient characteristics in the SDO database includes the surgical risk profile expressed with the EuroSCORE system.
As others have already pointed out [10], a key issue supporting the value of clinical databases is their greater face validity to health care providers and health professionals. Indeed, if the objective of clinical performance monitoring is the development of report cards to be made available to the public, the degree of trust that health professionals place on the source of information used for the assessment has a crucial role in making the initiative acceptable.
Nevertheless, although it is important, source of information is just one of the aspects affecting the credibility of performance monitoring initiatives for health professionals, and not necessarily the most relevant. Others include the extent to which the main objectives of these efforts are shared with health care providers, the relevance of the indicators used in monitoring clinical performance and, in general, the extent to which these efforts are perceived by health professionals to be supportive rather than punitive [15].
5. Conclusions
In ranking cardiac surgery centres according to their case-mix adjusted mortality rates, an administrative database including key clinical information provided the same results as a more complex and resource-demanding specialised database. However, as in the case of our study, showing that inexpensive administrative databases may reach the same conclusions as clinical databases, in labelling individual hospitals as ‘outliers’, does not mean that specialised clinical databases should necessarily be abandoned.
While there is a large consensus on the need for monitoring hospital performance, it should be noted that the choice of a source of information over another implicitly implies also a choice between different approaches to quality improvement and models of accountability [15]. In such a context, the main added value of clinical databases is the degree of direct participation of surgical teams required by their implementation. Such participation represents the necessary premise to expanding the scope of performance monitoring beyond the mere identification of ‘bad apples’ by some external agency, rather emphasising quality assessment as a specific responsibility of individual surgical teams.
Acknowledgements
Funding: support for this study was provided by Agenzia Sanitaria Regionale, Bologna (Italy).
References
Marshall M, Shekelle P, Brook R, Leatherman S. Dying to know: public release of information about quality of health care 2000;London (UK): Nuffield Trust and RAND.
Treasure T. Mortality in adult cardiac surgery. Br Med J 2005; 330:489–490.
Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997; 127:666–674.
Nobilio L, Fortuna D, Vizioli M, Berti E, Guastaroba P, Taroni F, Grilli R. Impact of regionalisation of cardiac surgery in Emilia-Romagna, Italy. J Epid Community Health 2004; 58:97–102.
Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999; 16:9–13.
Hosmer DW, Lemeshow S. Applied Logistic Regression 1989;New York: John Wiley & Sons.
Goldfield N, Pine M, Pine J. Measuring and managing health care quality. Procedures, techniques, and protocols 1991;Gaithersburg, MD: Aspen Publishers.
Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcomes measures and data sources. Am J Pub Health 1994; 84:1609–1614.
Geraci JM, Johnson ML, Gordon HS, Petersen NJ, Shroyer AL, Grover FL, Wray NP. Mortality after cardiac bypass surgery. Prediction from administrative versus clinical data. Med Care 2005; 43:149–158.
Ghali WA, Rothwell DM, Quan H, Brant R, Tu JV. A Canadian comparison of data sources for coronary artery bypass surgery outcome report cards. Am Heart J 2000; 140:402–408.
Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough. Health Serv Res 1997; 31:659–678.
Landon B, Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan YD. Judging hospitals by severity-adjusted mortality rates: the case of CABG surgery. Inquiry 1996; 33:155–166.
Hannan EL, Kilburn Jr H, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery. Does it matter. Med Care 1992; 30:892–907.
Shanmigam G, Wets M, Berg G. Additive and logistic EuroSCORE performance in high risk patients. Interact CardioVasc Thorac Surg 2005; 4:299–303.
Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001; 72:2155–2168.(Daniela Fortuna, Maria Vi)